nOCUMENT RESUME 



ED 293 876 



TM Oil 466 



AUTHOR 
TITLE 



PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Cloud-Si Iva, Connie; Denton, Jon J. 

The Development of a Low-Inference Observation 

Instrument To Assess Instructional Performance of 

Teaching Candidates. 

Apr 88 

42p.; Paper presented at the Annual Meeting of the 
American Educational Research Association (New 
Orleans, LA, April 5-9, 1988). 
Speeches/Conference Papers (150) — Reports - 
Research/Technical ( 143 ) 

MF01/PC02 Plus Postage. 

^Classroom Observation Techniques; Elementary 
Secondary Education; Evaluation Methods; Interrater 
Reliability; ^Minimum Competencies; Reliability; 
*Teacher Evaluation; Teacher Supervision; Teaching 
Skills; Validity 
*Teacher Candidates 



ABSTRACT 

A prototype low inference observation instrijunent to 
measure minimal teaching competencies of teaching candidates was 
deductively developed. Focus is on determining if observers could be 
trained to use the observation instrument with a high degree of 
reliability and validity. The instrument, entitled Classroom 
Observation and Assessment Scale for Teaching Candidates (COAST), was 
developed from apprai&al systems of the states of Florida, Georgia, 
and Texas, The initial draft was field-tested, and a panel of 11 
experts in teacher evaluation reviewed the indicators. The final form 
of the instrument covered: (1) instructional plannin'^; (2) classroom 
management and organization; (3) instructional techniques and 
materials; (4) communication and presentation techniques; (5) 
motivation of students; and (6) assessment of students. The 
instrument was tested on 27 teacher candidates (categorized in three 
grade level subdivisions of grades 1-12) with seven observers (six 
graduate students and one university faculty member). Inter-rater 
agreement was acceptable. Results suggest that COAST is an instrument 
that can be used across subject areas and grade levels with a high 
degree of content validity. Seven tables are presented, and an 
outline of the COAST Instrument Indicators is appended. (SLD) 



************** 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 

******** ilr********ilr** *************************************************** 



ERIC 



^ The Development of a LGW-mfejence Obeervation Instrunent 

^ to Assess instructional perfonna^ce of Teaching Qndidates 

ro 
o 



Qmnie Cloud-Silva and Jon J. Denton 
Texas AIM university 



U 8 DEPAirrMCNT Of EDUCATION 
Offtce ci Educational ReMarch and impfovement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER 'ERIC) 

XThjs document has been reproduced as 
received from the person or organization 
originating it 

D Minor changes have been made to improve 
reproduction quality 



• Pomts o( view or opinions Stated tn this docu 
ment do not necesaanly represent oMiciai 
OERl Position or > )ltcy 



• PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) " 



paper presented at the annual neeting of the American B*icational 
Research Association, New Orleans, Louisiana, April 6, 1988 



^ o 2 BEST COPY AVAILABLE 



Ibe Developnent of a Low - Inference Observation mstruBOit 
to ABseiis Instructional Performance of Teaching Ondidates 



Intense efforts have been made at the national and state levels to 
develop conprehensive observation instruments based on empirical research 
findings. The state of Texas, for example, has had to respond to 
legislation tl'.at requires the State Board of Education to adopt an appraisal 
process and to establish criteria to be used in the appraisal of classroom 
teachers. This system consists of a generic observation instrument used 
with all subject area and grade level teachers. Because the appraisal 
system is based on the premise that teachers consider evaluation systems 
that are clearly linked to teacher effectiveness literature as more fair and 
credible (Knapp, 1982), the instrument is comprised of teaching behaviors 
that have been found to be valid and reliable predictors of student 
achievement (Texas Bducation Code, Section 13.302(b). 

Purpose of the Study 

The purpose of the present study was to develop a prototype, low- 
inference observation instrument deductively derived from teacher 
effectiveness research that would measure minimal teaching competencies of 
teaching candidates. The study was designed to determine if observers could 
be trained to use the observation instrument with a high degree of 
reliability and validity. It was thought that the findings of the study 
might also provide an indication of how well candidates exhibit 
instructional behaviors required of inservice teachers. 

The research questions that guided the development of the observation 
instrument included: (1) Will the instrument be characterized by a high 



degree of content and face validity? (2) Can different observers use the 
instrument with a high degree of criterion-related agreement? (3) Can an 
observer use the observation instrument on different occasions with a high- 
degree of intraobserver agreement? (4) Will applications of the instrument 
reveal a high degree of interobserver agreement? and (5) Will different 
contextual conditions, such as subject area and grade level, affect the 
gcneralizability of the instrument? 

MenCD OF IBB SfffXK 
Initial uevel oMKUil o£ the Instnaent 

The development of the observation instrument, entited Classroom 
Observation and Assessment Scale for Teaching Candidates (COAST), was 
comprised of several steps. The first step wus to examine existing 
observation systems and teacher effectiveness literature to identify low- 
inference teaching behaviors associated with gains in student achievement. 
The appraisal system for the states of Florida (Florida Performance 
Measurement System, 1983) and Georgia (Teacher Performance Assessment 
Instrument, 1984), and preliminary drafts of the Texas appraisal system 
(Texas Teacher Appraisal System, 1986) gave direction to the format and 
structure of the instrument. To simplify the task of examining and 
structuring the content of the literature, research findings were 
categorized into six domains of teaching behaviors. Teaching behaviors 
representative of each domain were identified and served as indicators of 
teaching performance on the observation instrument. Behavioral descriptions 
and specific teaching examples for each of the indicators vere developed. 
This comprised the intial draft of the instrument. 

The intial draft was then field-tested at both the elementary and 
secondary levels with a variety of different subject areas. Field testing 



served a dual purpose: first, to investigate the occurrence of the 
behaviors in classrocms; anri second, to assist in the completion of a list 
of indicators and indicator dsscriptions that could be used in determining 
content validity. 

Jtadqes* Deteprinatioo of Content vari.idity 

A panel of 15 judges were asked to evaluate the indicators and the 
indicator descripticais. Eleven of the 15 agreed to participate. Of these 
eleven, five were experts in teacher effectiveness literature and six were 
Texas school administrators active in evaluating inservice teachers. Judges 
were asked to indicate if they agreed or disagreed that each specific 
indicator was important to the teaching act, and also to rate the wording of 
each indicator as clear or unclear. A criterion level of .75 was 
established for placement of indicators on the observation instrument prior 
to receiving feedback from the judg^* Ihe indicators received high ratings 
in terms of importance and clarity. In response to written suggestions made 
by the judges, redundant indicators were removed, the length of the 
instrument was reduced, and some of the behavioral descriptions of the 
indi::ators were rewritten in an attempt to remove ambiguous terminology and 
to mere clearly define the behaviors. 
Pinal Fora of the Instnncnt 

The observation instrument in its final form was comprised of si*: 
domains of teaching behaviors: (a) Instructional Planning; (b) Classroom 
Management and Organization; (c) Instructional Techniques and Materials; 
(d) communication and Presentation Techniques; (e) Motivation of Students; 
and (f) Assessment of Students. The domains of Classroom Management and 
Organization and Instructional Techniques and Materials were subdivided into 
three teaching behavior subdomains. Each of the domains and the subdomains 
included five teaching behaviors, or indicators, except for the Lesson Cycle 



- mstructional Behaviors subdomain, which had ten, and the Assessment of 
Students domain which had two behaviors. 

OCAST is a structured sign system designed to determine the presoice or 
absence of 62 low-inference teaching indicators in a teaching episode*. 
Directions for use of the instrument indicate that observers are to check 
instrument indicators as they occur during a lesson. A tally is made at the 
conclusion of a lesson of the indicators that did occur. The stipulation 
was made in the coding procedures that if a behavior did not occur during 
the lesson because of the inappropriateness of the b^avior to the lesson, 
this was to be indicated by the observer. 

Prooedures of the study 

Training Obeervers 

Videotapes wre made of the actual instructional sequences of three 
elementary teadiers. All three tapes were coded by the instrument developer 
(the criterion person) prior to the training session. A training manual, 
consisting of an explanation of the purpose of the study, behavioral 
descriptions of the indicators, specific teaching examples of the 
indicators, and the procedures to follow in using the instrument to code 
instructional classroom processes, was developed and discussed with the 
seven observers who participated in the study. 

One practice tape was used to demonstrate specific examples of the 
instrument indicators and to aid in the explanation of the indicators. 
Observers then watched the second practice videotape and practiced coding 
without direct assistance from the instrument developer. Coding tabulations 
were compared to those of the instrument developer (who also served as the 
criterion person in the study) and clarification was given as necessary. 
The observers then viewed the third and last training tape, and their coding 



was again compared to that of the criterion person. Because criterion-level 
agreement on videotapes representative of actual classrooms should be .15 or 
higher (Frick & Semmel 1978; Medley & Norton, 1971), this level was 
established prior to training and training was to continue until this level 
was reached by each of the observers. 

Ihe seven observers who participated in the study included six graduate 
students and one faculty member froin a major Southwestern university. All 
seven observers had had prior experience in evaluating preservice teadiers. 
Fcxir of the observers participated in the study for two semesters and three 
participated in the study for <x\e semester. 
RBcmtfent of Teaching Capdidates 

A total of 27 teaching candidates volunteered to be observed four times 

x 

during the course of their student teaching semester. Two of the 
observations were to be simultaneously coded by the criterion person and an 
observer. For the purpose of analyzing the data, and to test the 
generalizability of the instrument, teaching candidates were categorized 
into three grade-level subdivisions and represented different subject areas. 
The grade level subdivisions included grades 1-5, 6-8, and 9-12. Fourteen 
teaching candidates represented the basic skill areas of mathematics nnd 
language arts, and the other 13 teaching candidates represented other 
subject areas. 

Statistical Design of the Study 
Measures were taken to answer the research questions posed in the 
study. These questions dealt with content validity, interobserver 
agreement, intraobserver agreement, criterion-related agreement, and 
generalizability of COAST. Content validity was established by having a 
panel of eleven judges evaluate the instrument indicators with respect to 



thelL' importance and clarity. 

Criterion-related agreement, defined as the percentage of agreement 
between the observers and the criterion'^erson, was calculated using 
videotaped and actual classroom data. Four of the observers collected data 
across two semesters and coded the training tape before, during, and after 
the observation cycle. These three codings were compared to the codings of 
the criterion person. Three of the obsirvers participated in the study for 
one semester and coded the training tape before and after data collection. 
These two codings were also compared to codings of the criterion person. 

Intraobserver agreement, defined as the consistency between observer 
coding on separate occasions of the same videotape of actual classroom 
sequences, was also calculated. Intraobserver agreement of the four 
d>8ervers who participated across two semesters was determined by comparing 
their codings before, during, and after the data collection. The 
intraobserver agreement obtained by the three observers who participated for 
one semester was determined by comparing the codings made before and after 
data collection. 

Interobserver agreement, also known as interrater agreement, was also 
determined in this study in an effort to examine the extent to which the 
disagreement between the observers and the criterion person limited the 
reliability of the instrument. Interobserver agreement was determined by 
calculating the agreement between each observer's coding and the criterion 
persons coding on two simultaneously coded classroom observations for each 
of the 27 teaching candidates. 

The generalizability of the observation instrument was determined by 
comparing the observer agreement coefficients obtained in each of the three 
grade level subdivisions and the two subject area categories. The frequency 
of instrument indicator occurrence for each grade level subdivision and 
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subject are category was also calculated to investigate if some behaviors 
occurred more often at some grade levels and/or with certain subject areas. 

All observer agreement measures were calculated using Scott ^s (1955) 
coefficient. Scott^'s coefficient is one of the percent agreement indices 
that has been used to evaluate the interrater consistency of many teacher 
and student variables at m&ny grade levels (Anderson, Evertson, & Brophy, 
1978; Good & Grouws, 1977). Scott^'s coefficient is appropriate for 
calculating reliability estimates because the method is suitable for 
calculating observer agreement on the total categories of an observation 
instrument, comparing two observers'* codings at a time. The method is also 
unaffected by low frequencies across categories and this was particularly 
suitable for the data analyses in this study. 

Table 1 provides an outlay of the study indicating the data collection 
procedures and data treatment procedures that were utilized to answer the 
research questions posed in the study. 

BESOiaS OP THE STUDY 

A criterion level of .75 was established prior to receiving responses 
from the judges. All behaviors reached this level regarding their 
importance to the teaching act. Agreement levels ranging between 91% and 
100% were found for 51 of the 60 indicators. Hie level of agreement was 82% 
on the other nine indicators. Pertaining to clarity of the indicators, 49 
of the 60 indicators had an agreement level of 91% to 100%. Nine of the 
indicators had 82% agreement, and two of the indicators had 73%. 

CSianges were made in the instilment based on the judges'* scoring of the 
indicators and their comments regarding the behavioral descriptions of the 
indicators. Efforts were made to reduce the len<^'th of the instrument, to 
eliminate redundant indicators, and to eliminate indicators that required 



TabU 1 

Outlay of Data collection Procedures and Data Treatnent Procedures 



Heseorch Questions 



Data Collection 



Treatnent uf the Data 



11 - 111 11 the InstruMnt be 
characterlted by a high 
degree of content validity ? 



f2 - Can different observers use 
the InstruiNent with a high 
viegree of criterion-related 
agreement? 



Defers the Data Collection. 
A panel of 15 judges. 



Before^ during^ and after 
data collection by using 
videotaped eitaif>les of actual 
classrooa processes. Before 
training^ the criterion-level 
of Intraobserver agreeiwnt i#as 
set at .75. 



Decisions rules Hade prior 
to receiving feedback froA 
the Judges. Criterion-level 
Sft at .75. 



Used &cott*s coefficient to 
determine percentage of 
agreea«nt. Coders must reach 
a ailnliial level agreenent of 
.75. 



•3 - Can an observer use the observation 
Instruinent on different occasions 
vlth a high degree of Intraobserver 
a greement ? 



Videotapes of actual classroom 
processes used during training 
vere coded on three separate 
occasions. 



Used Scott's coefficient to 
determine percentage of 
agreement. 



14 - Will a test of the Instrument reveal 
a high degree of Intraobserver 
agreement ? 



Observer and criterion person 
ratings vere gathered after 
data collection. 



Used Scott's coefficient 
to determine the level of 
agreement between the ratings 
of the observers and the 
criterion person. 



15 - Will different contettual conditions^ 
such as subject-matter area and grade 
levels affect the generallgablllty of 
the Instrument? 



Observer and criterion person 
ratings were gathered after 
data collection for each of 
the three grade level divisions 
and the subject-matter areas. 



(1) Comparisons of Inter*- 
observer agreement 
coefficients across grade 
levels and subject areas 
was made. 



(3) Percentages of occurrence 
of each beha^'lor across 
grade level divisions and 
subject-matter areas was 
calculated. 
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observers to make high-level inferences. Although changes were made in the 
isntrument indicators , most of the changes involved clarification of 
indicators and behavioral descriptions of the indicators. The substance of 
the indicators remained basically unchanged, as the judges' ratings clearly 
reflected a high level of content validity. Table 2 presents the percentage 
of agreement of each of the instrument indicators. 

Criterion-related agreement was calculated on all seven observers 
before data collection began (following training) and after the data 
collection. Criterion-related agreement was also calculated between the two 
semesters for the four observers who collected data both semesters. The 
criterion-related agreement on the videotapes exceeded .80 for each of the 
seven observers, which has been considered an acceptable level of accuracy 
for sign systems (Frick & Semmel, 1978). Table 3 presents the criterion- 
related agreement. 

Bie results shown in Table 4 indicate that the intraobserver agreement 
measures on the videotaped tests demonstrated acceptable observer 
consistency and compared favorably with the consistency of observers 
reported in other experimental studies (Frick & Semmel, 1978; Medley & 
Norton, 1971). Hie observers were also consistent among themselves in their 
coding, and their levels of agreement indicated that their coding did not 
deteriorate during the data collection process. 

Interobserver agreement was determined by having the seven observers 
code simultaneously with the criterion-person on two separate occasions in 
each of the 27 teaching candidates' classrooms. There was a different 
number of simultaneously coded classroom observations for each of the seven 
observers. This occurred because the observers, who were also serving as 
student teacher supervisors, had different numbers of teaching candidates 
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rimleht VulicJiiy of Inaicu»eiit Indicaioco 



I «.|io fiance 



Indlc«toc8 



Clarity 



A<|ll'c: lliitdijlt't* 



CJeas Unclear 



J 

lout 

I not 



9t 

Ot 

Ui 
Ot 



2.0 ClassrooB Hanagencnt and Or9anlxatlon 

2.1 Hiinafjeaiehl. of Inuiiuctloiial Tine 

2.1.1 becjina class i>roaiptly 

2.1.2 engages students In Icainlng activities 
for a aiajorlty of the available class tlM 

2.1.3 routines are established for recurring 
act ivit ies 

2.1.4 provides sufficient tlaie for students to 
couplets Instructional activities 

2.1.5 nukes c»ooth transitions between activities 



loot 

7 It 
91t 

loot 

9lt 



Ot 

27t 

n 
ot 

9t 



lout 

H21 

02t 

<Jlt 
9it 



ot 

I lit 

lUt 

91 
9t 



2.2 Instruct* jiidl Organization of the Classioos 

2.2.1 nuteridls and facilities are ready for 
at the start of the lcc6or> 

2.2.2 ar raik<juRit'iitu lot ueatlng/grouping are 
appropriate foi iiiul iiictional activities 

2.2.3 uclicduleu for uuuigniieiiis are wade 
available to uiudcniti 

2.2.4 icvlbcu schedules ou neceusary 

2.2.5 arranijeo cl atiurooni for a variety of 
intiif uci ional puiposes 



loot 

62t 

02t 

62t 
73t 



Ot 

lot 

lUt 

IHt 
27t 
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Table 2 (Contiiiued) 



2.3 Manatjenent of Seatwork/lloRiework 

1U0% 01 2.3.1 giveb cleat directions for seatwork/ 100* 0% 

hoiiiewoik asslgnnenls 

lUlU 01 2.3.2 indlcales expected completion tine 100% 0% 

for assignments 

mot 01 2.3.3 circulates and pcovides assistance to 91% 9% 

btudeiits 

tjii 9% 2.3.4 provides ^ variety of iManingful '* 

seatwork/hoMiewock assigniients 

U2i 16% 2.3.5 provides opportunitieb for students 02% 10% 

to interact while co»pletin9 assignwents 

2.4 ^lanageaient of Student Conduct , 

SU ' 9% 2.4.1 establishes classroon nanagenent procedures 91% 9% 

appcopriate for the students in the 
c lassroom 

91% 9% 2.4.2 adninistero established rules and procedures 91% 9% 

for classrooM behavior 

1011% 0% 2.4.3 mohltors classrooiN activities 100% 0% 

H2% )li% 2.4.4 reinforces appropriate clacsroon behavior 91% 9% 

l>2% )U% 2.4.5 establishes physical arrangenent of the 02% llit 

clausrooii to piomote good behavior 
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Ydi'lc 2 iCoiili lined) 



I out 
lUUI 
IIHU 



til 
01 



3.0 JnatcuctlonaJ TeclinlqueB and N«t«cl«lB 

3.1 .J 9«inii Attention oC studentv to begin 

Inutcuctlun 

3. J .2 iicovldctt lint Iciifnioiy cc 1/9! vet focua of 
Iciiiion 

3.1.3 llnlitt Jeason to iiievloua ex|H!clencea 

3.1.4 detciMliica ctactlng |iolnta of atudenta 

3.1.5 pcuvldeu leuson olijectives to atudenta 



9U 

loot 

121 

loot 



9% 
9t 
01 

im 
01 



Hint 

loot 
loot 

u^t 

I oot 

loot 

1 001 

I oot 
I oot 
loot 



ot 

ot 
ot 

lut 
ot 

ot 

ot 

01 

ot 
ot 



3.2 I.eiitton Cycle - InatcuctW yaI Deliavluca 

3.2.1 iireacntu accucnte atlMuJua infucaiation ami 
iiutcciala 

3.2.2 luovldea Jearnei <juld«nc« of new infocaiatlon 

3.2.3 check a atudent under atandlng of new 
Infocnat Ion 

3.2.4 addreiicea specific needa of all atudenta 

3.2.5 check a tlie wofk and/of leaponaea of all 

fiiudent a 

3.2.6 |icovl(l(.*o ouided pidctlco of new JeacnliMjo 
by utudentii 

3.2.7 pruvldea Independent practice of new 
leiiriiliiiju by utudt-iii 

J. 2. 8 piovldeu feedback ciiiiccCMlmj biudent 
I et;p<»nuett 

3.2.9 leieiicbeii/ieKiediiitctt during Jeacon If 
iieeeiibiicy 

3.2.10 ccviewtt and tiu«i»dii2cu iiifuctiaiijn In 
cluiiOie of leLbun 



91% 

91% 
91% 

91% 
100% 

100% 

100% 

100% 

100% 

100% 



9% 

9% 
9% 

9% 
0% 

0% 

01 

0% 

0% 

0% 
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3.3 Utte of Instcuctlon«l Mateclal 








Gl 


3.1.1 uutB MiecUls appcopclatt tot the 
attMlnawnt of lecbon objectives 




» 9% 


1 mn 


Ul 


3.3.2 |»covl(l«s directions foe use of the 
ihbtcuctiofidl amtecials 


100% 


0% 


911 


9% 


3.3.3 tiwM audio-vlbual cesoucces effectively 


91% 


9% 




0% 


i.).4 uues 6U|>|»le»cnt«cy Material/ 9oes beyond 
leHt wlicn necc/'icicy 


91% 


Oft 

9% 


*Jll 


^% 


3.3.S i^covldes sufficient quantities of vateclals 
4.0 CjftMjnication and Presentation Techniques 


100% 


Aft 
Ul 


1 mn 


01 


4.1.1 utics correct bynt in oral discourse 


100% 


Aft 
111 


|0U« 


01 


4.1.2 uses correct syntai and spellin<| in written 
cowMiinication 


1 AAft 

IUU% 


1 0% 
lift 




9% 


4.1.3 vaiies voice characteclstics durln9 delivery 
of lestton 


a 1 ft 

9! % 






lot 


4.1.4 coainunicates effectively with special 
leaineis In the classroon 


821 


IttI 


1110% 


01 


4.1.5 eii|>hasites iii|>ortant fioints in lesson 


91% 


9% 


luol 


01 


4.1.6 encourayefi studentii to cipress ideas and 

ihlei eiitB 


100% 


0% 


IIMI% 


ot 


4.1.7 t:oiiiiaunicate8 enthusiaun for subject »attec 


9!% 


9% 


100% 


01 


4. 1.0 ciccuptt2 and incorporates ideas of students 
into the presentation 


100% 


0% 


lOO'i 


0% 


4.1.9 vaiies »o<le of presentation 


41% 


9% 


1 00% 


0% 


4.1.10 (leMOiisir at eti apiirupriate nonveibal 
coMiiinicat ioii 


91% 


9% 
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Table 2 (Coitiiiucd) 



5.0 Hotlvatlon of Students 

iOU% 0% 5.1 .1 provides options for students and/or allows 82% 18% 

utudeiits to choose and sake options 

IIMU 0% 5.1.2 provides opportunities for students to 100% 0% 

initiate Ideas, discussion, and activities 

10(1% 0% 5.1 .3 creates a lelaxed, confortable ataosphere 91% 9% 

for learning 

<j|'i «j% 5.1 .4 shows appr Gelation for Individual 

dif feruncefi 

loo« 0% 5.1.5 provides opportunities for students to 100% 0% 

share and display work efforts 

6.0 Acadealc Assessnent of Students 

liiuu 0% 6.1.1 provides foraatlve feedback to students on 100% • 0i% 

oral and written tertts, asslgoMnts, 

hor'»work and classwork 

U2t Jtt% 6.1.2 use^ student perforiiance on tests and 

assi'jnments 

U2% 1U% 6.1.3 uses a variety of assessnent techniques to 91% 9% 

deteralne the learning patterns of students 

iau% 0% 6.1.4 uses evaluation techniques appropriate for 91% 9% 

stated objectives 

911 9% 6.1.5 provides relnforceaent and feedback for 100% 01 

learning efforts ct stwderila 
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Tabic 3 

Criterion - Related Agrtement o£ the Cbservation Instrument 



Cefore During After 

Data Data Data 

Observers Colle ction Collection Collection 

1 .85 .98 .86 

2 .88 .91 .87 

3 .85 .93 .91 

4 .88 .88 .94 

5 .91 .93 
.88 .85 

7 .86 .92 



Table 4 

Zntraobserver Agrceaent 



During After 

Data Data 

Observers Collection Collection 

X .90 .90 

2 .85 .85 

3 .87 .85 

4 .91 .90 

5 .92 

6 .95 

7 .93 



who volunteered to participate in the study. Table 5 provides the 
interobserver coefficients for each of the seven observers. The 
interobserver agreement coefficients for eadti of the sev^ observers were as 
follows: CXDserver #1 » .91; Observer #2 « .89; Observer #3 » .85; Observer 
#4 » .92; Observer #5 » .94; Observer #6 « .90, and Observer #7 « .82. The 
total average of interobserver agreement was .89, indicating that different 
observers can use the instrument with a high degree of consistency and 
skill. 

Two steps were taken to determine the generalizability of the 
instrument. First, the levels of interobserver agreement were compared 
across grade levels and subject matter areas. Second, frequency of 
indicator occurrence across grade levels and subject are: categories was 
calculated. As previously mentioned, there were three grade level 
subdivisions, grades 1*5, 6*8, and 9-12, and also two subject area 
categories, the basic skill areas of ntathematics and language arts and 
"other" subject areas. 

At the elementary level, grades 1*5, five teaching candidates were 
observed teaching mathematics or language arts; two were observed teaching 
social studies; and two were observed teaching science. At the junior high 
level, grades 6*8, four teaching candidates were observed teaching 
mathematics or language arts; two were observed teching social studies; and 
one was observed teaching physical science; and one other was observed 
teaching history. At the high school levels grades 9*12, five teaching 
candidates were observed teaching mathematics or language arts; two were 
observed teaching government; two were observed teaching biology; and one 
was observed teaching chemistry. 

Table 6 presents the interobserver agreement coefficients for each of 
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Table 5 

Intcrobserver Agrccaient of Simultaneously Coded Ob8ervjtion& 



Observers 



1 


2 


3 


.98 


.74 


.92 


.97 


.88 


.91 


.91 


.96 


.88 


.06 


1.00 


.81 


.09 


.88 


.87 


.91 


.84 


.35 Avg. 


.89 


.91 


.78 


.86 


.94 


.80 


.87 




.73 


.01 


'Avg.= .89 


.80 


.80 




.79 


.92 




.94 


.87 




.94 


.94 




.89 


.95 






1.00 


Avg.« 


.85 



Avg.= .91 



4 


5 


6 


7 


.89 


.92 


.86 


.85 


.98 


.91 


.84 


.84 


.87 


.93 


.94 


.77 


.94 


.99 


.94 


.82 


.92 Avg.° 


.94 Avg.= 


.90 Avg.= 


.82 



Total Average » .89 
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Table 6 

Intccobsccver Agreement and Avemges Across Grade Levels and Subject Areas 

Mathematics and Language Arts Other Subject Areas 



* 1 2 3 4 5 Avg. 6 7 8 9 Avg. 

,85 .92 .80 .92 .98 = .89 .85 .85 .87 .98 « .88 



10 11 12 13 Avg. 14 15 16 17 Avg. 

.90 .89 .85 .88 » .88 .86 .85 .79 .79 = .82 



18 19 20 21 22 Avg. 23 24 25 26 27 Avg. 

.90 .87 .89 .94 .93 = .91 98 .96 .94 .91 .92 = .94 



* Note. Numbers 1 to 27 represent the 27 teaching candidates being observed (hiring 
the study. 

ERIC 21 



00 



the three grade level subdivisions and also for the two subject categories. 
The coefficients were relatively high and consistent across the grade level 
subdivisions and subject areas. At the elemr-^'-ary subdivision, the average 
interobserver agreement for areas of mathematics and language arts was .89, 
and .88 for the subject areas of health and social studies. At the junior 
high level, the average interobserver agreement coefficient was ,88 for 
mathematics and language arts, and «82 for the subject areas of social 
studies, physical science, and history. Interobserver agreement 
coefficients were higher at the high school level as ,90 was obtained for 
mathematics and language arts and .94 was obtained for the subject areas of 
government, chemistry, and biology. 

The second measure waken to determine the generalizability of COAST 
dealt with determining the frequency with which the instrument indicators 
occurred across subject aueas and grade levels. Frequency percentages 
indicate that, overall, the consistency of indicator occurrence was stable 
across the contextual settings respresented in the study. These frequency 
percentages are shown in Table 7. 

Frequency results did suggest, however, that a few of the indicators 
appeared more at some grade levels than at others. Secondary teaching 
candidates, for example, did not reinforce students for good classroom 
behavior as frequently as did teaching candidates at either the elementary 
or junior high level. Secondary teaching candidates provided students with 
more options to initiate and discuss ideas and activities than did 
elementary or junior high teaching candidates. Teaching candidates at the 
elementary and junior high levels used audio-visual resources less 
effectively and used supplementary materials (materials outside of the 
textbook) less frequently than did teaching candidates at the secondary 
level. 
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Table 7 

frequency o£ Initrunent Ind icator Occurrence 
Across Grade Levels and Subject Areas 



1.0 INSTRUCYIONAL PLANNXKG 

1.1 Flannlng-Developed In Written Lesson Plans 



Instrument Indicators 





1.1.1 


1.1.2 


1.1.3 


1.1.4 


1.1.5 


Grades 1-5 












Hath and Lang. Arts 


69% 


94% 


94% 


94% 


56% 


Other Subject Areas 


81% 


100% 


94% 


100% 


100% 


Avg. 


75% 


97% 


94% 


97% 


78% 


Grades €-8 












Kath and Lang* Arts 


66% 


94% 


94% 


94% 


86% 


Other Subject Areas 


56% 


83% 


56% 


75% 


67% 


Avg. 


73% 


69% 


76% 


65% 


76% 


Grades 9-12 












Kath and Lang. Arts 


92% 


92% 


63% 


92% 


83% 


ether Subject Areas 


c6% 


68% 


100% 


6 9% 


100% 
















90% 


90% 


92% 


61% 


92% 



Table 7 (Continued) 



1.2 Planning Implemented 



Instrument Indicators 



Graces 1-S 


1.2.1 


1 ,2.2 


1.2.3 


1.2.4 


1.2. 












Math and Lang. Arts 


56% 


88% 


81% 


69% 


36% 


Other Subjsct Areas 


81% 


100% 


81% 


94% 


86% 


Avg. 


69% 


94% 


81% 


82% 


63% 


Graces 6-6 












Kath and Lang. Arts 


ei% 


94% 


81% 


54% 


81% 


Other Subject Areas 


50% 


75% 


58% 


67% 


50% 


Avg. 


66% 


85% 


70% 


8n 


66% 


Grades 9-12 












Hath and Lang. Arts 


83% 


75% 


75% 


67% 


75% 


Other Subject Areas 


69% 


81% 


94% 


63% 


100% 


Ave. 


76 1 


78% 


85% 


63% 


38% 



Table 7 (Continued) 



2.0 CLASSRCOff FMIAGFMENT AND ORGANIZATION 

2.1 Hanagtment nstructional Ti»e 



Instrument Indicator! 



Grades 1-5 


2.1.1 


2.1.2 


2.1.3 


2.1.4 


2.1. 


Nath and Lang. Arts 


1001 


951 


951 


951 


601 


Other Subject Areas 


941 


1001 


811 


1001 


88% 


Avg. 


971 


951 


88% 


981 


64% 



Grad*8 6-8 



Kath and tang 


. Arts 


86% 


94% 


61% 


75% 


75% 


Other Subject 


Areas 


81% 


61% 


75% 


75% 


75% 




Avg. 


85% 


88% 


78% 


75% 


75% 



Grades 9-12 



Nath and Lang 


. Ar ' 


100% 


95% 


90% 


80% 


85% 


Other Subject 


Areas 


100% 


100% 


95% 


85% 


100% 




Avg. 


100% 


98% 


93% 


83% 


93% 



2G 



Table 7 (Continued) 



2.2 Instructional Organization of the Classroom 



Instrument Indicators 



Grades 1-5 
Hath and Lang. Arts 


2.2.1 


2.2.2 


2.2.3 


2.2.4 


2.2. 


95% 


95% 


100% 


55% 


70% 


Other Subject Areas 


100% 


66% 


94% 


3 8% 


13% 


Avg. 


98% 


92% 


97% 


47% 


42% 



Grades 6-8 



Kath and Lang 


• Arts 


100% 


94% 


61% 


69% 


19% 


Other Subject 


Areas 


66% 


61% 


66% 


44% 


13% 




Avg. 


94% 


66% 


65% 


57% 


16% 


Grades 9- 


12 












Kath and Lang 


• Arts 


95% 


90% 


95% 


65% 


5% 


Other Subject 


Areas 


100% 


100% 


95% 


80% 


50% 




Avg . 


98% 


95% 


95\ 


73% 


26% 



T«ble 7 (Continued) 



2.3 nanagesent of Student Conduct 



Instrument Indicators 



Grades 1*5 


2.3.1 


2.3.2 


2.3.3 


2.3.4 


2.3. 


Hath and Lang* Arts 


loot 


95t 


90t 


70t 


90t 


Other Subject Areas 


loot 


94t 


loot 


75t 


63t 


Avg. 


loot 


95t 


9St 


73t 


77t 



Grades 6-8 



Hath and Lang 


• Arts 


94t 


81t 


75t 


S6t 


75t 


Other Subject 


Areas 


eit 


81t 


75t 


sot 


63t 




Avg. 


88t 


81t 


75t 


S3t 


69t 



Grades 9-12 



Hath and Lang. 


Arts 


80t 


90t 


80t 


sot 


80t 


Other Subject 


Areas 


90t 


loot 


80t 


25t 


8St 




nvg. 


85t 


95t 


80t 


38t 


83t 



Table 7 (Continued) 



3.0 INSTBUCTIONAL TECHNIQUES AND MATERIALS 

3.1 Establishes Learning Set 



Instrunient Indicators 



Grades 1-5 


3.1.1 


3.1.2 


3.1.3 


3.1.4 


3.1. 












Hath and Lang. Arts 


80t 


70t 


90t 


90t 


40t 


Other Subject Areas 


75t 


81t 


loot 


81t 


50t 


Avg. 


78t 


75t 


95% 


86 1 


45t 


Grades 6-8 












Hath and Lang. Arts 


75t 


75t 


loot 


63t 


44t 


Other Subject Areas 


69t 


50t 


81 1 


63t 


25t 


Avg. 


72t 


63t 


91t 


63% 


35t 


Grades 9-12 












Hath and Lang. Acts 


95t 


95t 


loot 


85t 


60t 


Other Subject Areas 


65% 


95t 


95t 


75t 


6St 


Avg. 


90t 


95% 


9et 


80% 


63t 
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Tablt 7 (Continued) 



3.2 tttson Cyclt - I nitructlonal B»h avloci 

XnstCMB«nt Indicators 



3.2.1 3.2.2 3.2.3 3.2.4 3.2. S 
Gcadts 1-5 — — — — — — " 



Hatn and Lan^. Acts 


95% 


95% 


95% 


90% 


90% 


Oth«r Subject Artas 


81% 


94% 


68% 


100% 


88% 


Av9. 


S8% 


95% 


92% 


95% 


89% 



Gradas 8-8 



Kath and tan^. Acts 


100% 


94% 


94% 


94% 


94% 


Othar Subjact Acaas 


63% 


75% 


89% 


63% 


75% 


Av9. 


82% 


85% 


82% 


79% 


85% 



Grades 9-12 



Kath and tan^. Acts 


100% 


100% 


90% 


90% 


80% 


Other Subject Aceas 


90% 


100% 


100% 


100% 


90% 


Av9. 


95% 


100% 


95% 


95% 


85% 
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Table 7 (Continued) 



3.3 Use of Instructional Material 

Instrument Indicators 



Grades 


3.3.1 


3.3.2 


3.3.3 


3.3.4 


3.3.5 












Hath and Lang. Arts 


90% 


80% 


45% 


65% 


95% 


Other Subject Areas 


94% 


81% 


56% 


69% 


81% 


Avg. 


92% 


81% 


51% 


67% 


68% 


Grades S-8 












Kath and Lang. Acts 


100% 


81% 


43% 


S6% 


56% 


Other Subject Areas 


81% 


56% 


63% 


69% 


75% 


Avg. 


91% 


69% 


53% 


63% 


66% 


Grades 9-12 












Kath and Lang. Arts 


90% 


75% 


75% 


70% 


60% 


Other Subject Areas 


100% 


7 5% 


75% 


85% 


100% 


Avg. 


95% 


75% 


75% 


7 8% 


80% 



n 
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Tablt 7 (Continctd) 



4.0 COHNUIIICATICN MID PRE SENT AT I ON TECBNIIOCS 



InstruMnt Indicators 



lirMCS 


4.1 


4.2 


4.3 


4.4 


4.S 


H«th and Lan9. Arts 


90% 


95% 


60% 


75% 


70% 


Otbtr Subjtct Artas 


94% 


94% 


88% 


81% 


81% 


Av9. 


92% 


9S% 


74% 


78% 


76% 


Gr«dts <-l 












Rath and Lan9. Arts 


100% 


100% 


81% 


100% 


69% 


Othtr Subject Artas 


7S% 


7S% 


63% 


so% 


63% 


Av9. 


18% 


18% 


72% 


7S% 


66% 


Cradas 9-12 












Matl. and Lan9. Arts 


100% 


100% 


8S% 


6S% 


80% 


Cthtr Subjtct Artas 


100% 


100% 


80% 


80% 


8S% 


Avq. 


loot 


100% 


83% 


73% 


83% 




Table 7 (Continued) 



S.O nOTlVATIOM FOR LEARNING 



Instrument Indicators 



Gradas 1-S 


5.1 


5.2 


S.3 


5.4 


5.5 


Hath and Lan9. Arts 


2S% 


60% 


95% 


85t 


80% 


Other Subject Araas 


31% 


7S% 


94% 


94% 


81% 


nV9. 


28% 


68% 


95% 


90% 


81% 


Grades 6-6 












Math and Lan9. Arts 


38% 


63% 


94% 


94% 


75% 


Othar Subjtct Areas 


19% 


S0% 


56% 


56% 


31% 


Av9. 


"29% 


57% 


7 5% 


75% 


53% 


Gradts 9-12 












Kath and Lan9* Arts 


S% 


9S% 


90% 


80% 


70% 


Othtr Subjtct Areas 


40% 


80% 


80% 


70% 


50% 


Ave . 


23% 


88% 


85% 


75% 


60% 
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Table 7 (Continued) 



€•0 ACADEMIC A5S£S5KCNT OT STUDCNTS 



Instrument Indicators 

6.1 €.2 

Cades 1-5 — 

ttath and Lang. Arts f4% 81% 

ether Subject Areas 100% 94% 

Av^. f7% 88% 

Grades 6-8 

Kath and Lang. Arts 1C0% 100% 

Other Subject Areas 83% 75% 

Avg. 92% 88% 

Grades 9-12 

Math and Lan9. Arts 1C0% 83% 

ether Subject Areas 100% 94% 

Avg. 100% 89% 



Frequency of occurrence of a few of the instrument indicators was 
consistently low across subject area categories and grade levels. The 
teaching candidates did not arrange their classrooms to accommodate a 
variety of instructional purposes. Their classroom arrangements were 
inflexible and static throughout the semester. Teaching candidates also did 
not provide learners with instructional objectives in the set inductions of 
their lessons. Another indicator that was conistently low in frequency was 
the provision of a review or summary of main points in the conclusion of the 
lesson. These teaching candidates also infrequently provided students with 
options during their lessens. 

Results did indicate, however, that these 27 teaching candidates were 
proficient in exhibiting several competency skills. The teaching candidates 
appeared to have been good instructional planners; good managers of the 
utilization of instructional tirie; effective organizers of instruction in 
that materials and equipment were ready at the start of the lesson and the 
seating/grouping arrangements were appropriate for instructiorial activities; 
and good managers of student conduct in that classroom management procedures 
were established and administered and student conduct was monitored. These 
teaching candidates used correct syntax and spelling in oral and written 
discourse, communicated enthusiasm, and demonstrated appropriate nonverbal 
behaviors. 

Overall, teaching candidates were also effective in utilizing the 
instructional techniques in the Lesson Cycle * Instructional Behaviors 
subdomain. Accurate stimulus information was presented, key features of the 
lessons were pointed out, guided practice and independent practice 
opportunities were provided, clear directions for assignments were given, 
and feedback concerning student performance was provided. The teaching 
candidates' classrooms were comfortable atmospheres for learning and 
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teaching candidates personalized instruction by recognizing individual 
differences of students. Frequency percentages also indicated that these 
teaching candidates were proficient in providing assessment of student 
learning, either formal or informal, that was appropriate for the lesson 
content • 

GONCLDSIONS AMD RBGOMIDIDATZONS 

Collectively, the results to the research questions suggest that OOAST 
is a generic observation instrument that can be used across subject areas 
and grade levels. The instrument has a high degree of content validity, and 
it can be used by observers with a high degree of accuracy and consistency. 
Based on the frequency of indicator occurrence across the small number of 
subject areas and grade levels represented in the study, the instrument 
appears to be generalizable across different contextual conditions. This 
conclusion requires further testing as some of the indicators were rather 
low in frequency. Further testing might determine if this low frequ^cy was 
due to the proficiency level of the teaching candidates participating in the 
study, or the inappropriateness of the indicators on a generic observation 
instrument. 

The findings of the study also indicate that the instrument can be used 
to evaluate lessons that fit the direct-instruction model of teaching. 
Forms completed by the observers concerning the grouping structures and 
teaching methods utilized in the observed lessons indicated that out of the 
total 108 lessons observed, 107 lessons utilizeo a whole-class grouping 
structure for at least part of the lesson. Eight lessons utilized a small- 
group structure that was teacher-led, and seven of ttiese eight lessons were 
in elementary reading group situations. Itiere were no lessons observed that 
incorporated student-to-student interaction or allowed students to work 
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together to complete instructimal tasks. Most of these lessons appeared to 
have incorporated the direct instruction model of teaching which limits the 
types of instructional strategies observed during this study. 

The findings of the study do indicate that COAST may be suitable for 
inclusion in teacher preparatioi programs. Because COkSI is comprised of 
behaviors identified by research as effective and because these behaviors 
are often included on observation instruments that are used to evaluate 
inservice teachers, an instrument such as CX)AST might be valuable in the 
assessment of teaching candidates. 

COAST could be used in a teacher preparation program to improve 
teaching candidate performance. Teaching candidates could first be pre* 
tested over the behaviors incorporated in CX3AST, then provided in-depth 
instruction on each of the instrument indicators, then assessed in actual 
classrooms with COAST to determine if teaching performance improved as a 
result of the training. Results obtained by using OOAST could be used to 
diagnose specific areas of teaching that need to be remediated in the 
teaching candidate'^s preparation. 

^averal recc^mendations for further study are made. First, this study 
should be replicated in other situations to increase the total number of 
teaching candidates observed with COAST. By having larger numbers of 
teaching candidates observed, the generalizability of the instrument could 
be further verified. Second, replicatioi studies should focus on specific 
content areas to determine how well teaching candidates from just one 
subject area compare in their evaluations. This would provide more 
information concerning the appropriateness of instrument indicators for 
evaluating teacher performance in specific subject areas. Third, a 
replication study should be conducted that extends the number of subject 
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areas and grade levels represented. The number of teaching candidates at 
different grade levels and subject areas, as well as specific grade levels 
and subject areas, were limited in this study, and by extending these more 
information should be obtained concerning the external validity and 
generalizability of the instrument. And fourth, a replication study should 
be conducted that addresses a common criticism raised about the ability of 
observation instruments to adequately and fairly assess teacher performance 
when different instructional strategies are utilized. A deliberate attenpt 
should be made in a replication study to assess a wide range of 
instructional strategies to determine if significantly different ratings 
will be obtained from those obtained in direct instruction lessons. 
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INSD«]HEMT INDICATORS OF GQAST 



1.0 msmomoAL planning 

1.1 Developed in Lesson Plans 

1.1.1 lesson plans specify the set induction of the lesson 

1.1.2 lesson plans specify the instructional materials to be used 

1.1.3 lesson plans specify the guided practice techniques to be used 

1.1.4 lesson plans specify the independent practice techniques 

1.1.5 lesson plans specify the assessment techniques to be used 



1.2 iBplemented as Planned 

1.2.1 the set induction of the lesson is inplemented as planned 

1.2.2 specified instructional materials are used during lesson 

1.2.3 specified guided practice techniques are used during lesson 

1.2.4 specified independent practice techniques are used 

1.2.5 specified assessment techniques arae inplemented during lesson 



2.0 CLASSROOM MANA5EMEMI AND CR5ANIZATI0N 



2.1 Management of Instrucitonal Time 

2.1.1 begins class promptly 

2.1.2 engages students in learning activities for a majority of 
the available classtime 

2.1.3 routines are established for recurring activities 

2.1.4 provides sufficient time for students to complete instructional activities 

2.1.5 makes smooth transitions between activities 



2.2 Instructional Organization of the Classroom 

2.2.1 instructional materials are ready for use at the start of the lesson 

2.2.2 instructional materials and equipment are ready for use at the start 
of ^he lesson 

2.2.3 arrai^gements for seating/grouping are appropriate for instructional 
activities 

2.2.4 schedules for assignments are m&de available to students 

2.2.5 arranges room for a variety of instructional purposes 
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2.3 Wanaqqaent of Student Conduct 

2.3.1 establishes classroom management procedures appropriate for the 
students in the classroom 

2.3.2 administers established rules and procedures for classr^ ^. behavior 

2.3.3 mc^itors student conduct 

2.3.4 reinforces appropriate classroom behavior 

2.3.5 establishes physical arrangement of classroom to promote good 
classroom behavior 



3.0 INSIRDCnOmL TaCBNIQDES AND MMrOUALS 



3.1 establishes Leanung Set 

3.1.1 gains attention of students to begin instruction 

3.1.2 provides anticipatory set/gives focus of lesson 

3.1.3 links lesson to previous experiences 

3.1.4 determines starting points of students 

3.1.5 provides lesson objectives to students 



3.2 t^fesson cycle - Instnictignaj. Bdiaviors 

3.2.1 presents stimulus information and materials 

3.2.2 points out key features of new information 

3.2.3 provides for guided practice of new information by students 

3.2.4 provides for independent practice of new information by students 

3.2.5 gives clear directions for seatwork/horoework assignments 

3.2.6 indicates expected coirpletion time for assignments 

3.2.7 circulates and provides assistance to students 

3.2.8 provides feedback concerning student performance 

3.2.9 reteaches/remediates during lesson if necessary 

3.2.10 reviews/suirmarizes information in closure of lesson 



3.3 Dse of instructional Material 

3.3.1 uses materials appropriate for the attainment of lesson objectives 

3.3.2 pro/ides explanations and directions to students for use of the 
material 

3.3.3 uses audio-visual resources effectively 

3.3.4 uses supplementary material/goes beyond text 

3.3.5 provides sufficient quantities of materials for the class 
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4.0 OQMIDNICATION AND FRESQlTMnON TGCSNIQDES 

4.1 uses correct syntax in oral discourse 

4.2 uses correct syntax and spelling in written materials 

4.3 connunicates enthusiasm for the subject matter 

4.4 varies mode of presentation 

4.5 demonstrates appropriate nonverbal behaviors 



5.0 MOnvmON pgr I£ABNING 

5.1 provides <^ions to students during lesson/allows students to 
take responsibility for their lesson 

5.2 provides qpportunities for students to initiate ideas, discussion, 
and/or activities 

5.3 creates a relaxed, comfortable atmosphere for learning 

5.4 shows appreciation for individual differences 

5.5 provides c^jportunities for students to share ideas and work efforts 



6.0 iODEMIC ASSESSMBIT AND EVAZJOAXION 1SCHNIQDES 

6.1 evaluates student understanding either formally or informally during 
the lesson 

6.2 evaluation technqiues used are appropriate for the content of the lesson 
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